Receive queue device with efficient queue flow control, segment placement and virtualization mechanisms

ABSTRACT

A mechanism for offloading the management of receive queues in a split (e.g. split socket, split iSCSI, split DAFS) stack environment, including efficient queue flow control and TCP/IP retransmission support. An Upper Layer Protocol (ULP) creates receive work queues and completion queues that are utilized by an Internet Protocol Suite Offload Engine (IPSOE) and the ULP to transfer information and carry out send operations. As consumers initiate receive operations, receive work queue entries (RWQEs) are created by the ULP and written to the receive work queue (RWQ). The ISPOE is notified of a new entry to the RWQ and it subsequently reads this entry that contains pointers to the data that is to be received. After the data is received, the IPSOE creates a completion queue entry (CQE) that is written into the completion queue (CQ). After the CQE is written, the ULP subsequently processes the entry and removes it from the CQE, freeing up a space in both the RWQ and CQ. The number of entries available in the RWQ are monitored by the ULP so that it does not overwrite any valid entries. Likewise, the IPSOE monitors the number of entries available in the CQ, so as not overwrite the CQ.

RELATED APPLICATION

[0001] This application is related to commonly assigned and co-pendingU.S. patent application Ser. No. ______ (Attorney Docket No.AUS920020123US1) entitled “SPLIT SOCKET SEND QUEUE APPARATUS AND METHODWITH EFFICIENT QUEUE FLOW CONTROL, RETRANSMISSION AND SACK SUPPORTMECHANISMS”, filed on ______, U.S. patent application Ser. No. ______(Attorney Docket No. AUS920020129US1) entitled “MEMORY MANAGEMENTOFFLOAD FOR RDMA ENABLED NETWORK ADAPTERS”, filed on ______, U.S. patentapplication Ser. No. ______ (Attorney Docket No. AUS920020127US1)entitled “iSCSI DRIVER TO ADAPTER INTERFACE PROTOCOL”, filed on ______,and co-pending and commonly assigned U.S. patent application Ser. No.10/132,461 (Attorney Docket No. AUS920020065US1) entitled “LOGICALPARTITION HOSTED VIRUTAL INPUT/OUTPUT USING SHARED TRANSLATION CONTORLENTRIES”, all of which are hereby incorporated by reference.

BACKGROUND OF THE INVENTION

[0002] 1. Technical Field

[0003] The present invention relates generally to communicationprotocols between a host computer and an input/output (I/O) device. Morespecifically, the present invention provides a hardware implementationfor offloading management of a receive queue. In particular, the presentinvention provides a mechanism by which work requests are turned intowork queue entries (WQEs) and are passed from Upper Layer Protocol (e.g.sockets) software to an Internet Protocol (IP) Suite Offload Engine(IPSOE). The present invention also provides a mechanism by whichcompleted WQEs are passed back to the Upper Layer Protocol (ULP)software. The present invention also provides a mechanism for supportingSelective Acknowledgements. Finally, the present invention provides amechanism by which an IPSOE can be shared between virtual hosts of asingle physical host.

[0004] 2. Description of Related Art

[0005] In an Internet Protocol (IP) Network, the software provides amessage passing mechanism that can be used to communicate withinput/output devices, general purpose computers (host), and specialpurpose computers. The message passing mechanism consists of a transportprotocol, an upper level protocol, and an application programminginterface. The key standard transport protocols used on IP networkstoday are the Transmission Control Protocol (TCP) and the User DatagramProtocol (UDP). TCP provides a reliable service and UDP provides anunreliable service. In the future the Stream Control TransmissionProtocol (SCTP) will also be used to provide a reliable service.Processes executing on devices or computers access the IP networkthrough upper level protocols, such as Sockets, iSCSI, and Direct AccessFile System (DAFS).

[0006] Unfortunately, the TCP/IP software consumes a considerable amountof processor and memory resources. This problem has been coveredextensively in the literature (see J. Kay, J. Pasquale, “Profiling andreducing processing overheads in TCP/IP”, IEEE/ACM Transactions onNetworking, Vol 4, No. 6, pp.817-828, December 1996; and D. D. Clark, V.Jacobson, J. Romkey, H. Salwen, “An analysis of TCP processingoverhead”, IEEE Communications Magazine, volume: 27, Issue: 6, June1989, pp 23-29). In the future the network stack will continue toconsume excessive resources for several reasons, including: increaseduse of networking by applications; use of network security protocols;and the underlying fabric bandwidths are increasing at a higher ratethan microprocessor and memory bandwidths. To address this problem, theindustry is offloading the network stack processing to an IP SuiteOffload Engine (IPSOE).

[0007] There are two offload approaches being taken in the industry. Thefirst approach uses the existing TCP/IP network stack, without addingany additional protocols. This approach can offload TCP/IP to hardware,but unfortunately does not remove the need for receive side copies. Asnoted in the papers above, copies are one of the largest contributors tocentral processing unit (CPU) and memory bandwidth utilization. Toremove the need for copies, the industry is pursuing the second approachthat consists of adding Framing, Direct Data Placement (DDP), and RemoteDirect Memory Access (RDMA) over the TCP and SCTP protocols. The IPSuite Offload Engine (IPSOE) required to support these two approaches issimilar, the key difference being that in the second approach thehardware must support the additional protocols.

[0008] The IPSOE provides a message passing mechanism that can be usedby sockets, Internet Small Computer System Interface (iSCSI), DirectAccess File Systems (DAFS), and other Upper Layer Protocols (ULPs) tocommunicate between nodes. Processes executing on host computers, ordevices, access the IP network by posting send/receive messages tosend/receive work queues on an IPSOE. These processes also are referredto as “consumers”.

[0009] The send/receive work queues (WQ) are assigned to a consumer as aqueue pair (QP). The messages can be sent over three different transporttypes: traditional TCP, RDMA TCP, UDP, or SCTP. Consumers retrieve theresults of these messages from a completion queue (CQ) through IPSOEsend and receive work completion (WC) queues. The source IPSOE takescare of segmenting outbound messages and sending them to thedestination. The destination IPSOE takes care of reassembling inboundmessages and placing the inbound messages in the memory space designatedby the destination's consumer. These consumers use IPSOE verbs to accessthe functions supported by the IPSOE. The software that interprets verbsand directly accesses the IPSOE is known as the IPSO interface (IPSOI).

[0010] Today the host CPU performs most IP suite processing. IP SuiteOffload Engines offer a higher performance interface for communicatingto other general purpose computers and I/O devices. Data sends orreceives through the IPSOE require that the CPU either copy data fromone memory location to another or register the memory so that the IPSOEcan directly access the memory region. Each of these options requiressignificant CPU resources with the memory registration option beingpreferred for large memory transfers, however, as network speedsincrease the amount of CPU resources required will increase. A simplemechanism is needed to implement Receive Queue in the IPSOE and performRDMA, DDP, framing, and TCP/IP processing in the IPSOE. The mechanismneeds to maintain all RDMA, DDP, framing, TCP, IP, and Ethernet state inthe IPSOE. It must also provide the necessary protection to support outof user space Receive Queue operations. The present invention alsoprovides a mechanism for supporting Selective Acknowledgements. Finally,the present invention provides a mechanism by which an IPSOE can beshared between virtual hosts of a single physical host.

SUMMARY OF THE INVENTION

[0011] The present invention provides a method, computer programproduct, and distributed data processing system for management of areceive queue in a split (e.g. split socket, split iSCSI, split DAFS)stack in order to reduce the processing overhead in host processors.

[0012] Specifically, the present invention is directed to a mechanismfor turning work requests into work queue entries and inserting thesework queue entries into the receive queue. This invention also providesa mechanism by which work queue entries are transmitted from Upper LayerProtocol (e.g. socket) software to the Internet Protocol Suite OffloadEngine (IPSOE) and are processed by the IPSOE. The present inventionalso provides a mechanism by which the IPSOE converts completed workqueue elements into work completion entries and passes these workcompletion entries back to the software. The present invention alsoprovides a mechanism for supporting Selective Acknowledgements. Finally,the present invention provides a mechanism by which an IPSOE can beshared between virtual hosts of a single physical host.

BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The novel features believed characteristic of the invention areset forth in the appended claims. The invention itself, however, as wellas a preferred mode of use, further objectives and advantages thereof,will best be understood by reference to the following detaileddescription of an illustrative embodiment when read in conjunction withthe accompanying drawings, wherein:

[0014]FIG. 1 is a diagram of a distributed computer system illustratedin accordance with a preferred embodiment of the present invention;

[0015]FIG. 2 is a functional block diagram of a host processor node inaccordance with a preferred embodiment of the present invention;

[0016]FIG. 3A is a diagram of an IP Suite Offload Engine in accordancewith a preferred embodiment of the present invention;

[0017]FIG. 3B is a diagram of a switch in accordance with a preferredembodiment of the present invention;

[0018]FIG. 3C is a diagram of a router in accordance with a preferredembodiment of the present invention;

[0019]FIG. 4 is a diagram illustrating processing of work requests inaccordance with a preferred embodiment of the present invention;

[0020]FIG. 5 is a diagram illustrating a portion of a distributedcomputer system in accordance with a preferred embodiment of the presentinvention in which a TCP or SCTP transport is used;

[0021]FIG. 6 is an illustration of a data frame in accordance with apreferred embodiment of the present invention;

[0022]FIG. 7 is a diagram illustrating a portion of a distributedcomputer system in accordance with a preferred embodiment of the presentinvention;

[0023]FIG. 8 is a diagram illustrating the network addressing used in adistributed networking system in accordance with the present invention;

[0024]FIG. 9 is a diagram of a portion of a distributed computer systemcontaining subnets in a preferred embodiment of the present invention;

[0025]FIG. 10 is a diagram of a layered communication architecture usedin a preferred embodiment of the present invention;

[0026]FIG. 11 is an exemplary diagram that depicts the contents of thesocket context entry created for a given socket in accordance with thepresent invention;

[0027]FIG. 12 is a diagram of an exemplary Work Queue Element list anddetails of the Work Queue Element Entries in accordance with the presentinvention;

[0028]FIG. 13 is a diagram of an exemplary Completion Queue Context anddetails of the Completion Queue Element Entries in accordance with thepresent invention;

[0029]FIG. 14 is a flowchart outlining an exemplary operation forcreation of a queue pair in accordance with the present invention;

[0030]FIG. 15 is a flowchart outlining an exemplary operation of areceive operation in accordance with the present invention;

[0031]FIG. 16 is an exemplary diagram illustrating a receive queuedoorbell mechanism for informing the IPSOE of a receive work queue entrycount;

[0032]FIG. 17 is an exemplary diagram illustrating a completion queuedoorbell mechanism for informing the IPSOE of completion queue credits;

[0033]FIGS. 18A and 18B illustrate a diagram and flowchart,respectively, depicting the initialization of the SelectiveAcknowledgement Table in accordance with the present invention;

[0034]FIG. 19 is a flowchart outlining the Selective Acknowledgementprocessing performed by the IPSOE on incoming TCP/IP Segment isprocessed in accordance with the present invention;

[0035]FIG. 20 is an exemplary diagram of the operations supported by theIPSOE and the tables used to maintain state for those operations inaccordance with the present invention;

[0036]FIG. 21 is an exemplary diagram of the IPSOE MAC Table and theHosting Partition Resource Mapping Table, which is used by the HostingPartition to map between physical IPSOE Resources and Virtual IPSOEResources, in accordance with the present invention;

[0037]FIG. 22 is a flowchart outlining the IPSOE VirtualizationInitialization mechanism in accordance with the present invention;

[0038]FIG. 23 is a flowchart outlining the Hosted Server Creationmechanism in accordance with the present invention;

[0039]FIG. 24 is a flowchart outlining the Hosted Server Operation Trapsto Hosting Partition mechanism in accordance with the present invention;

[0040]FIG. 25 is a flowchart outlining the IPSOE Management Verbs (Open,Query, Modify, and Close) mechanism in accordance with the presentinvention;

[0041]FIG. 26 is a flowchart outlining the CQ Management Verbs (Create,Query, Modify, and Destroy) mechanism in accordance with the presentinvention;

[0042]FIG. 27 is a flowchart outlining the QP Management Verbs (Create,Query, Modify, and Destroy) mechanism in accordance with the presentinvention;

[0043]FIG. 28 is a flowchart outlining the Memory Management Verbs (AllRegisters, All Reregisters, All Allocates, All Deregisters) mechanism inaccordance with the present invention;

[0044]FIG. 29 is a flowchart outlining the PD and IP Address AliasManagement Verbs (Allocate PD, Deallocate PD) mechanism in accordancewith the present invention;

[0045]FIG. 30 is a flowchart outlining the Post Send and Post ReceiveVerbs mechanism in accordance with the present invention;

[0046]FIG. 31 is a flowchart outlining the Poll CQ, Set CQ Handler, SetAsync Event Handler Verbs mechanism in accordance with the presentinvention; and

[0047]FIG. 32 is a flowchart outlining the IPSOE Incoming Ethernet FrameProcessing mechanism in accordance with the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

[0048] The present invention provides a distributed computing systemhaving endnodes, switches, routers, and links interconnecting thesecomponents. The endnodes can be Internet Protocol Suite Offload Enginesor traditional host software based Internet protocol suites. Eachendnode uses send and receive queue pairs to transmit and receivemessages. The endnodes segment the message into frames and transmit theframes over the links. The switches and routers interconnect theendnodes and route the frames to the appropriate endnode. The endnodesreassemble the frames into a message at the destination.

[0049] With reference now to the figures and in particular withreference to FIG. 1, a diagram of a distributed computer system isillustrated in accordance with a preferred embodiment of the presentinvention. The distributed computer system represented in FIG. 1 takesthe form of an Internet protocol network (IP net), such as IP net 100and is provided merely for illustrative purposes and the embodiments ofthe present invention described below can be implemented on computersystems of numerous other types and configurations. For example,computer systems implementing the present invention can range from asmall server with one processor and a few input/output (I/O) adapters tomassively parallel supercomputer systems with hundreds or thousands ofprocessors and thousands of I/O adapters. Furthermore, the presentinvention can be implemented in an infrastructure of remote computersystems connected by an Internet or intranet.

[0050] IP net 100 is a high-bandwidth, low-latency networkinterconnecting nodes within the distributed computer system. A node isany component attached to one or more links of a network and forming theorigin and/or destination of messages within the network. In thedepicted example, IP net 100 includes nodes in the form of hostprocessor node 102, host processor node 104, and redundant arrayindependent disk (RAID) subsystem node 106. The nodes illustrated inFIG. 1 are for illustrative purposes only, as IP net 100 can connect anynumber and any type of independent processor nodes, storage nodes, andspecial purpose processing nodes. Any one of the nodes can function asan endnode, which is herein defined to be a device that originates orfinally consumes messages or frames in IP net 100.

[0051] In one embodiment of the present invention, an error handlingmechanism in distributed computer systems is present in which the errorhandling mechanism allows for TCP or SCTP communication between endnodesin a distributed computing system, such as IP net 100.

[0052] A message, as used herein, is an application-defined unit of dataexchange, which is a primitive unit of communication between cooperatingprocesses. A frame is one unit of data encapsulated by Internet ProtocolSuite headers and/or trailers. The headers generally provide control androuting information for directing the frame through IP net 100. Thetrailer generally contains control and cyclic redundancy check (CRC)data for ensuring frames are not delivered with corrupted contents.

[0053] Within a distributed computer system, IP net 100 contains thecommunications and management infrastructure supporting various forms oftraffic, such as storage, interprocess communications (IPC), fileaccess, and sockets. IP net 100 shown in FIG. 1 includes a switchedcommunications fabric 116, which allows many devices to concurrentlytransfer data with high-bandwidth and low latency in a secure, remotelymanaged environment. Endnodes can communicate over multiple ports andutilize multiple paths through the IP net fabric. The multiple ports andpaths through the IP net shown in FIG. 1 can be employed for faulttolerance and increased bandwidth data transfers.

[0054] The IP net 100 in FIG. 1 includes switch 112, switch 114, androuter 117. A switch is a device that connects multiple links togetherand allows routing of frames from one link to another link using thelayer 2 destination address field. When the Ethernet is used as thelink, the destination field is known as the media access control (MAC)address. A router is a device that routes frames based on the layer 3destination address field. When Internet Protocol (IP) is used as thelayer 3 protocol, the destination address field is an IP address.

[0055] In one embodiment, a link is a full duplex channel between anytwo network fabric elements, such as endnodes, switches, or routers.Example suitable links include, but are not limited to, copper cables,optical cables, and printed circuit copper traces on backplanes andprinted circuit boards.

[0056] For reliable service types (TCP and SCTP), endnodes, such as hostprocessor endnodes and I/O adapter endnodes, generate request frames andreturn acknowledgment frames. Switches and routers pass frames along,from the source to the destination.

[0057] In IP net 100 as illustrated in FIG. 1, host processor node 102,host processor node 104, and RAID subsystem node 106 include at leastone IPSOE to interface to IP net 100. In one embodiment, each IPSOE isan endpoint that implements the IPSOI in sufficient detail to source orsink frames transmitted on IP net 100. Host processor node 102 containsIPSOEs in the form of host IPSOE 118 and IPSOE 120. Host processor node104 contains IPSOE 122 and IPSOE 124. Host processor node 102 alsoincludes central processing units 126-130 and a memory 132interconnected by bus system 134. Host processor node 104 similarlyincludes central processing units 136-140 and a memory 142interconnected by a bus system 144.

[0058] IPSOE 118 provides a connection to switch 112, while IPSOE 124provides a connection to switch 114, and IP Suite Offload Engines 120and 122 provide a connection to switches 112 and 114.

[0059] In one embodiment, an IP Suite Offload Engine is implemented inhardware or a combination of hardware and offload microprocessor(s). Inthis implementation, IP suite processing is offloaded to the IPSOE. Thisimplementation also permits multiple concurrent communications over aswitched network without the traditional overhead associated withcommunicating protocols. In one embodiment, the IPSOEs and IP net 100 inFIG. 1 provide the consumers of the distributed computer system withzero processor-copy data transfers without involving the operatingsystem kernel process, and employs hardware to provide reliable, faulttolerant communications.

[0060] As indicated in FIG. 1, router 117 is coupled to wide areanetwork (WAN) and/or local area network (LAN) connections to other hostsor other routers. In this example, RAID subsystem node 106 in FIG. 1includes processor 168, memory 170, IP Suite Offload Engine (IPSOE) 172,and multiple redundant and/or striped storage disk unit 174.

[0061] IP net 100 handles data communications for storage,interprocessor communications, file accesses, and sockets. IP net 100supports high-bandwidth, scalable, and extremely low latencycommunications. User clients can bypass the operating system kernelprocess and directly access network communication components, such asIPSOES, which enable efficient message passing protocols. IP net 100 issuited to current computing models and is a building block for new formsof storage, cluster, and general networking communication. Further, IPnet 100 in FIG. 1 allows storage nodes to communicate among themselvesor communicate with any or all of the processor nodes in a distributedcomputer system. With storage attached to IP net 100, the storage nodehas substantially the same communication capability as any hostprocessor node in IP net 100.

[0062] In one embodiment, the IP net 100 shown in FIG. 1 supportschannel semantics and memory semantics. Channel semantics is sometimesreferred to as send/receive or push communication operations. Channelsemantics are the type of communications employed in a traditional I/Ochannel where a source device pushes data and a destination devicedetermines a final destination of the data. In channel semantics, theframe transmitted from a source process specifies a destinationprocesses' communication port, but does not specify where in thedestination processes' memory space the frame will be written. Thus, inchannel semantics, the destination process pre-allocates where to placethe transmitted data.

[0063] In memory semantics, a source process directly reads or writesthe virtual address space of a remote node destination process. Theremote destination process need only communicate the location of abuffer for data, and does not need to be involved in the transfer of anydata. Thus, in memory semantics, a source process sends a data framecontaining the destination buffer memory address of the destinationprocess. In memory semantics, the destination process previously grantspermission for the source process to access its memory.

[0064] Channel semantics and memory semantics are typically bothnecessary for storage, cluster, and general networking communications. Atypical storage operation employs a combination of channel and memorysemantics. In an illustrative example storage operation of thedistributed computer system shown in FIG. 1, a host processor node, suchas host processor node 102, initiates a storage operation by usingchannel semantics to send a disk write command to the RAID subsystemIPSOE 172. The RAID subsystem examines the command and uses memorysemantics to read the data buffer directly from the memory space of thehost processor node. After the data buffer is read, the RAID subsystememploys channel semantics to push an I/O completion message back to thehost processor node.

[0065] In one exemplary embodiment, the distributed computer systemshown in FIG. 1 performs operations that employ virtual addresses andvirtual memory protection mechanisms to ensure correct and proper accessto all memory. Applications running in such a distributed computersystem are not required to use physical addressing for any operations.

[0066] Turning next to FIG. 2, a functional block diagram of a hostprocessor node is depicted in accordance with a preferred embodiment ofthe present invention. Host processor node 200 is an example of a hostprocessor node, such as host processor node 102 in FIG. 1. In thisexample, host processor node 200, shown in FIG. 2, includes a set ofconsumers 202-208, which are processes executing on host processor node200. Host processor node 200 also includes IP Suite Offload Engine(IPSOE) 210 and IPSOE 212. IPSOE 210 contains ports 214 and 216 whileIPSOE 212 contains ports 218 and 220. Each port connects to a link. Theports can connect to one IP net subnet or multiple IP net subnets, suchas IP net 100 in FIG. 1.

[0067] Consumers 202-208 transfer messages to the IP net via the verbsinterface 222 and message and data service 224. A verbs interface isessentially an abstract description of the functionality of an IP SuiteOffload Engine. An operating system may expose some or all of the verbfunctionality through its programming interface. Basically, thisinterface defines the behavior of the host. Additionally, host processornode 200 includes a message and data service 224, which is ahigher-level interface than the verb layer and is used to processmessages and data received through IPSOE 210 and IPSOE 212. Message anddata service 224 provides an interface to consumers 202-208 to processmessages and other data.

[0068] With reference now to FIG. 3A, a diagram of an IP Suite OffloadEngine is depicted in accordance with a preferred embodiment of thepresent invention. IP Suite Offload Engine 300A shown in FIG. 3Aincludes a set of queue pairs (QPs) 302A-310A, which are used totransfer messages to the IPSOE ports 312A-316A. Buffering of data toIPSOE ports 312A-316A is channeled using the network layer's quality ofservice field (QOSF), for example, the Traffic Class field in the IPVersion 6 specification, 318A-334A. Each network layer quality ofservice field has its own flow control. Internet Engineering Task Force(IETF) standard network protocols are used to configure the link andnetwork addresses of all IP Suite Offload Engine ports connected to thenetwork. Two such protocols are Address Resolution Protocol (ARP) andDynamic Host Configuration Protocol. Memory translation and protection(MTP) 338A is a mechanism that translates virtual addresses to physicaladdresses and validates access rights. Direct memory access (DMA) 340Aprovides for direct memory access operations using memory 350A withrespect to queue pairs 302A-310A.

[0069] A single IP Suite Offload Engine, such as the IPSOE 300A shown inFIG. 3A, can support thousands of queue pairs. Each queue pair consistsof a send work queue (SWQ) and a receive work queue (RWQ). The send workqueue is used to send channel and memory semantic messages. The receivework queue receives channel semantic messages. A consumer calls anoperating system specific programming interface, which is hereinreferred to as “verbs”, to place work requests (WRs) onto a work queue.

[0070]FIG. 3B depicts a switch 300B in accordance with a preferredembodiment of the present invention. Switch 300B includes a packet relay302B in communication with a number of ports 304B through link ornetwork layer quality of service fields such as IP version 4's Type ofService field 306B. Generally, a switch such as switch 300B can routeframes from one port to any other port on the same switch.

[0071] Similarly, FIG. 3C depicts a router 300C according to a preferredembodiment of the present invention. Router 300C includes a frame relay302C in communication with a number of ports 304C through network layerquality of service fields such as IP version 4's Type of Service field306C. Like switch 300B, router 300C will generally be able to routeframes from one port to any other port on the same router.

[0072] With reference now to FIG. 4, a diagram illustrating processingof work requests is depicted in accordance with a preferred embodimentof the present invention. In FIG. 4, a receive work queue 400, send workqueue 402, and completion queue 404 are present for processing requestsfrom and for consumer 406. These requests from consumer 406 areeventually sent to hardware 408. In this example, consumer 406 generateswork requests 410 and 412 and receives work completion 414. As shown inFIG. 4, work requests placed onto a work queue are referred to as workqueue elements (WQEs).

[0073] Send work queue 402 contains work queue elements (WQEs) 422-428,describing data to be transmitted on the IP net fabric. Receive workqueue 400 contains work queue elements (WQEs) 416-420, describing whereto place incoming channel semantic data from the IP net fabric. A workqueue element is processed by hardware 408 in the IPSOE.

[0074] The verbs also provide a mechanism for retrieving completed workfrom completion queue 404. As shown in FIG. 4, completion queue 404contains completion queue elements (CQEs) 430-436. Completion queueelements contain information about previously completed work queueelements. Completion queue 404 is used to create a single point ofcompletion notification for multiple queue pairs. A completion queueelement is a data structure on a completion queue. This elementdescribes a completed work queue element. The completion queue elementcontains sufficient information to determine the queue pair and specificwork queue element that completed. A completion queue context is a blockof information that contains pointers to, length, and other informationneeded to manage the individual completion queues.

[0075] Example work requests supported for send work queue 402 shown inFIG. 4 are as follows. A send work request is a channel semanticoperation to push a set of local data segments to the data segmentsreferenced by a remote node's receive work queue element. For example,work queue element 428 contains references to data segment 4 438, datasegment 5 440, and data segment 6 442. Each of the send work request'sdata segments contains part of a virtually contiguous memory region. Thevirtual addresses used to reference the local data segments are in theaddress context of the process that created the local queue pair.

[0076] A remote direct memory access (RDMA) read work request provides amemory semantic operation to read a virtually contiguous memory space ona remote node. A memory space can either be a portion of a memory regionor portion of a memory window. A memory region references a previouslyregistered set of virtually contiguous memory addresses defined by avirtual address and length. A memory window references a set ofvirtually contiguous memory addresses that have been bound to apreviously registered region.

[0077] The RDMA read work request reads a virtually contiguous memoryspace on a remote endnode and writes the data to a virtually contiguouslocal memory space. Similar to the send work request, virtual addressesused by the RDMA read work queue element to reference the local datasegments are in the address context of the process that created thelocal queue pair. The remote virtual addresses are in the addresscontext of the process owning the remote queue pair targeted by the RDMAread work queue element.

[0078] A RDMA write work queue element provides a memory semanticoperation to write a virtually contiguous memory space on a remote node.For example, work queue element 416 in receive work queue 400 referencesdata segment 1 444, data segment 2 446, and data segment 448. The RDMAwrite work queue element contains a scatter list of local virtuallycontiguous memory spaces and the virtual address of the remote memoryspace into which the local memory spaces are written.

[0079] A RDMA FetchOp work queue element provides a memory semanticoperation to perform an atomic operation on a remote word. The RDMAFetchOp work queue element is a combined RDMA Read, Modify, and RDMAWrite operation. The RDMA FetchOp work queue element can support severalread-modify-write operations, such as Compare and Swap if equal. TheRDMA FetchOp is not included in current RDMA over IP standardizationefforts, but is described here, because it may be used as a value-addedfeature in some implementations.

[0080] A bind (unbind) remote access key (R_Key) work queue elementprovides a command to the IP Suite Offload Engine hardware to modify(destroy) a memory window by associating (disassociating) the memorywindow to a memory region. The R_Key is part of each RDMA access and isused to validate that the remote process has permitted access to thebuffer.

[0081] In one embodiment, receive work queue 400 shown in FIG. 4 onlysupports one type of work queue element, which is referred to as areceive work queue element. The receive work queue element provides achannel semantic operation describing a local memory space into whichincoming send messages are written. The receive work queue elementincludes a scatter list describing several virtually contiguous memoryspaces. An incoming send message is written to these memory spaces. Thevirtual addresses are in the address context of the process that createdthe local queue pair.

[0082] For interprocessor communications, a user-mode software processtransfers data through queue pairs directly from where the bufferresides in memory. In one embodiment, the transfer through the queuepairs bypasses the operating system and consumes few host instructioncycles. Queue pairs permit zero processor-copy data transfer with nooperating system kernel involvement. The zero processor-copy datatransfer provides for efficient support of high-bandwidth andlow-latency communication.

[0083] When a queue pair is created, the queue pair is set to provide aselected type of transport service. In one embodiment, a distributedcomputer system implementing the present invention supports three typesof transport services: TCP, SCTP, and UDP.

[0084] TCP and SCTP associate a local queue pair with one and only oneremote queue pair. TCP and SCTP require a process to create a queue pairfor each process that TCP and SCTP are to communicate with over the IPnet fabric. Thus, if each of N host processor nodes contains Pprocesses, and all P processes on each node wish to communicate with allthe processes on all the other nodes, each host processor node requiresP²×(N−1) queue pairs. Moreover, a process can associate a queue pair toanother queue pair on the same IPSOE.

[0085] A portion of a distributed computer system employing TCP or SCTPto communicate between distributed processes is illustrated generally inFIG. 5. The distributed computer system 500 in FIG. 5 includes a hostprocessor node 1, a host processor node 2, and a host processor node 3.Host processor node 1 includes a process A 510. Host processor node 3includes a process C 520 and a process D 530. Host processor node 2includes a process E 540.

[0086] Host processor node 1 includes queue pairs 4, 6 and 7, eachhaving a send work queue and receive work queue. Host processor node 2has a queue pair 9 and host processor node 3 has queue pairs 2 and 5.The TCP or SCTP of distributed computer system 500 associates a localqueue pair with one and only one remote queue pair. Thus, the queue pair4 is used to communicate with queue pair 2; queue pair 7 is used tocommunicate with queue pair 5; and queue pair 6 is used to communicatewith queue pair 9.

[0087] A WQE placed on one send queue in a TCP or SCTP causes data to bewritten into the receive memory space referenced by a receive WQE of theassociated queue pair. RDMA operations operate on the address space ofthe associated queue pair.

[0088] In one embodiment of the present invention, the TCP or SCTP ismade reliable because hardware maintains sequence numbers andacknowledges all frame transfers. A combination of hardware and IP netdriver software retries any failed communications. The process client ofthe queue pair obtains reliable communications even in the presence ofbit errors, receive underruns, and network congestion. If alternativepaths exist in the IP net fabric, reliable communications can bemaintained even in the presence of failures of fabric switches, links,or IP Suite Offload Engine ports.

[0089] In addition, acknowledgements may be employed to deliver datareliably across the IP net fabric. The acknowledgement may, or may not,be a process level acknowledgement, i.e. an acknowledgement thatvalidates that a receiving process has consumed the data. Alternatively,the acknowledgement may be one that only indicates that the data hasreached its destination.

[0090] The User Datagram Protocol is connectionless. The UDP is employedby management applications to discover and integrate new switches,routers, and endnodes into a given distributed computer system. The UDPdoes not provide the reliability guarantees of the TCP or SCTP. The UDPaccordingly operates with less state information maintained at eachendnode.

[0091] Turning next to FIG. 6, an illustration of a data frame isdepicted in accordance with a preferred embodiment of the presentinvention. A data frame is a unit of information that is routed throughthe IP net fabric. The data frame is an endnode-to-endnode construct,and is thus created and consumed by endnodes. For frames destined to anIPSOE, the data frames are neither generated nor consumed by theswitches and routers in the IP net fabric. Instead for data frames thatare destined to an IPSOE, switches and routers simply move requestframes or acknowledgment frames closer to the ultimate destination,modifying the link header fields in the process. Routers may modify theframe's network header when the frame crosses a subnet boundary. Intraversing a subnet, a single frame stays on a single service level.

[0092] Message data 600 contains data segment 1 602, data segment 2 604,and data segment 3 606, which are similar to the data segmentsillustrated in FIG. 4. In this example, these data segments form a frame608, which is placed into frame payload 610 within data frame 612.Additionally, data frame 612 contains cyclic redundancy check (CRC) 614,which is used for error checking. Additionally, routing header 616 andtransport header 618 are present in data frame 612. Routing header 616is used to identify source and destination ports for data frame 612.Transport header 618 in this example specifies the sequence number andthe source and destination port number for data frame 612. The sequencenumber is initialized when communication is established and incrementsby 1 for each byte of frame header, DDP/RDMA header, data payload, andCRC. Frame header 620 in this example specifies the destination queuepair number associated with the frame and the length of the Direct DataPlacement and/or Remote Direct Memory Access (DDP/RDMA) header plus datapayload plus CRC. DDP/RDMA header 622 specifies the message identifierand the placement information for the data payload. The messageidentifier is constant for all frames that are part of a message.Example message identifiers include, for example, send, write RDMA, andread RDMA.

[0093] In FIG. 7, a portion of a distributed computer system 700 isdepicted to illustrate an example request and acknowledgmenttransaction. Distributed computer system 700 in FIG. 7 includes a hostprocessor node 702 running process A 716 and a host processor node 704running process B 718. Host processor node 702 includes an IPSOE 706.Host processor node 704 includes an IPSOE 708. The distributed computersystem in FIG. 7 includes IP net fabric 710, which includes switch 712and switch 714. The IP net fabric includes a link coupling IPSOE 706 toswitch 712; a link coupling switch 712 to switch 714; and a linkcoupling IPSOE 708 to switch 714.

[0094] In the example transactions, host processor node 702 includes aclient process A. Host processor node 704 includes a client process B.Client process A interacts with host IPSOE 706 through queue pair 23 720comprising send queue 724 and receive queue 726. Client process Binteracts with host IPSOE 708 through queue pair 24 722 comprising sendqueue 728 and receive queue 730. Queue pairs 23 and 24 are datastructures that include a send work queue and a receive work queue.

[0095] Process A initiates a message request by posting work queueelements to the send queue of queue pair 23. Such a work queue elementis illustrated in FIG. 4. The message request of client process A isreferenced by a gather list contained in the send work queue element.Each data segment in the gather list points to part of a virtuallycontiguous local memory region, which contains a part of the message,such as indicated by data segments 1, 2, and 3, which respectively holdmessage parts 1, 2, and 3, in FIG. 4.

[0096] Hardware in host IPSOE 706 reads the work queue element andsegments the message stored in virtual contiguous buffers into dataframes, such as the data frame illustrated in FIG. 6. Data frames arerouted through the IP net fabric, and for reliable transfer services,are acknowledged by the final destination endnode. If not successfullyacknowledged, the data frame is retransmitted by the source endnode.Data frames are generated by source endnodes and consumed by destinationendnodes.

[0097] With reference to FIG. 8, a diagram illustrating the networkaddressing used in a distributed networking system is depicted inaccordance with the present invention. A host name provides a logicalidentification for a host node, such as a host processor node or I/Oadapter node. The host name identifies the endpoint for messages suchthat messages are destined for processes residing on an endnodespecified by the host name. Thus, there is one host name per node, but anode can have multiple IPSOEs.

[0098] A single link layer address (e.g. Ethernet Media Access LayerAddress) 804 is assigned to each port 806 of an endnode component 802. Acomponent can be an IPSOE, switch, or router. All IPSOE and routercomponents must have a MAC address. A media access point on a switch isalso assigned a MAC address.

[0099] One network address (e.g. IP Address) 812 is assigned to eachport 806 of an endnode component 902. A component can be an IPSOE,switch, or router. All IPSOE and router components must have a networkaddress. A media access point on a switch is also assigned a MACaddress.

[0100] Each port of switch 810 does not have a link layer addressassociated with it. However, switch 810 can have a media access port 814that has a link layer address 816 and a network layer address 808associated with it.

[0101] A portion of a distributed computer system in accordance with apreferred embodiment of the present invention is illustrated in FIG. 9.Distributed computer system 900 includes a subnet 902 and a subnet 904.Subnet 902 includes host processor nodes 906, 908, and 910. Subnet 904includes host processor nodes 912 and 914. Subnet 902 includes switches916 and 918. Subnet 904 includes switches 920 and 922.

[0102] Routers create and connect subnets. For example, subnet 902 isconnected to subnet 904 with routers 924 and 926. In one exampleembodiment, a subnet has up to 216 endnodes, switches, and routers.

[0103] A subnet is defined as a group of endnodes and cascaded switchesthat is managed as a single unit. Typically, a subnet occupies a singlegeographic or functional area. For example, a single computer system inone room could be defined as a subnet. In one embodiment, the switchesin a subnet can perform very fast wormhole or cut-through routing formessages.

[0104] A switch within a subnet examines the destination link layeraddress (e.g. MAC address) that is unique within the subnet to permitthe switch to quickly and efficiently route incoming message frames. Inone embodiment, the switch is a relatively simple circuit, and istypically implemented as a single integrated circuit. A subnet can havehundreds to thousands of endnodes formed by cascaded switches.

[0105] As illustrated in FIG. 9, for expansion too much larger systems,subnets are connected with routers, such as routers 924 and 926. Therouter interprets the destination network layer address (e.g. IPaddress) and routes the frame.

[0106] An example embodiment of a switch is illustrated generally inFIG. 3B. Each I/O path on a switch or router has a port. Generally, aswitch can route frames from one port to any other port on the sameswitch.

[0107] Within a subnet, such as subnet 902 or subnet 904, a path from asource port to a destination port is determined by the link layeraddress (e.g. MAC address) of the destination host IPSOE port. Betweensubnets, a path is determined by the network layer address (IP address)of the destination IPSOE port and by the link layer address (e.g. MACaddress) of the router port, which will be used to reach thedestination's subnet.

[0108] In one embodiment, the paths used by the request frame and therequest frame's corresponding positive acknowledgment (ACK) frame is notrequired to be symmetric. In one embodiment employing oblivious routing,switches select an output port based on the link layer address (e.g. MACaddress). In one embodiment, a switch uses one set of routing decisioncriteria for all its input ports in the switch. In one exampleembodiment, the routing decision criteria are contained in one routingtable. In an alternative embodiment, a switch employs a separate set ofcriteria for each input port.

[0109] A data transaction in the distributed computer system of thepresent invention is typically composed of several hardware and softwaresteps. A client process data transport service can be a user-mode or akernel-mode process. The client process accesses IP Suite Offload Enginehardware through one or more queue pairs, such as the queue pairsillustrated in FIGS. 3A, 5, and 8. The client process calls an operatingsystem specific programming interface, which is herein referred to as“verbs.” The software code implementing verbs posts a work queue elementto the given queue pair work queue.

[0110] There are many possible methods of posting a work queue elementand there are many possible work queue element formats, which allow forvarious cost/performance design points, but which do not affectinteroperability. A user process, however, must communicate to verbs ina well-defined manner, and the format and protocols of data transmittedacross the IP net fabric must be sufficiently specified to allow devicesto interoperate in a heterogeneous vendor environment.

[0111] In one embodiment, IPSOE hardware detects work queue elementpostings and accesses the work queue element. In this embodiment, theIPSOE hardware translates and validates the work queue element's virtualaddresses and accesses the data.

[0112] An outgoing message is split into one or more data frames. In oneembodiment, the IPSOE hardware adds a DDP/RDMA header, frame header andCRC, transport header and a network header to each frame. The transportheader includes sequence numbers and other transport information. Thenetwork header includes routing information, such as the destination IPaddress and other network routing information. The link header containsthe destination link layer address (e.g. MAC address) or other localrouting information.

[0113] If a TCP or SCTP is employed, when a request data frame reachesits destination endnode, acknowledgment data frames are used by thedestination endnode to let the request data frame sender know therequest data frame was validated and accepted at the destination.Acknowledgement data frames acknowledge one or more valid and acceptedrequest data frames. The requester can have multiple outstanding requestdata frames before it receives any acknowledgments. In one embodiment,the number of multiple outstanding messages, i.e. request data frames,is determined when a queue pair is created.

[0114] One embodiment of a layered architecture 1000 for implementingthe present invention is generally illustrated in diagram form in FIG.10. The layered architecture diagram of FIG. 10 shows the various layersof data communication paths, and organization of data and controlinformation passed between layers.

[0115] IPSOE endnode protocol layers (employed by endnode 1011, forinstance) include upper level protocols 1002 defined by consumer 1003,transport layer 1004; network layer 1006, link layer 1008, and physicallayer 1010. Switch layers (employed by switch 1013, for instance)include link layer 1008 and physical layer 1010. Router layers (employedby router 1015, for instance) include network layer 1006, link layer1008, and physical layer 1010.

[0116] Layered architecture 1000 generally follows an outline of aclassical communication stack in order to complete consumer operations1012 of transferring data between consumers 1003 and 1005. With respectto the protocol layers of endnode 1011, for example, upper layerprotocols 1002 employs verbs to create messages at transport layer 1004.Transport layer 1004 passes messages 1014 to network layer 1006. Networklayer 1006 routes frames between network subnets 1016. Link layer 1008routes frames within a network subnet 1018. Physical layer 1010 sendsbits or groups of bits to the physical layers of other devices. Each ofthe layers is unaware of how the upper or lower layers perform theirfunctionality.

[0117] Consumers 1003 and 1005 represent applications or processes thatemploy the other layers for communicating between endnodes. Transportlayer 1004 provides end-to-end message movement. In one embodiment, thetransport layer provides four types of transport services as describedabove which are traditional TCP, RDMA over TCP, SCTP, and UDP. Networklayer 1006 performs frame routing through a subnet or multiple subnetsto destination endnodes. Link layer 1008 performs flow-controlled 1020,error checked, and prioritized frame delivery across links.

[0118] Physical layer 1010 performs technology-dependent bittransmission. Bits or groups of bits are passed between physical layersvia links 1022, 1024, and 1026. Links can be implemented with printedcircuit copper traces, copper cable, optical cable, or with othersuitable links.

[0119] As discussed above, the present invention provides a mechanismfor managing a receive queue in a split stack in order to reduce theprocessing overhead in host processors. An Upper Layer Protocol (e.g.socket) library creates Work Queues (WQ) and Completion Queues (CQ) thatare utilized by an Internet Protocol Suite Offload Engine (IPSOE) andthe Upper Layer Protocol (ULP) to transfer information and carry outsend operations. As consumers initiate send operations, Work QueueEntries (WQE) are created by the ULP and written to the Send Work Queue(SWQ). The ISPOE is notified of a new entry to the SWQ and itsubsequently reads this entry, which contains pointers to the data thatis to be sent. After the data is sent and acknowledgements are received,the IPSOE creates a Completion Queue Entry (CQE) that is written to theCQ. The CQE includes a Work Request ID that associates a given WQE to aCQE. After the CQE is written, the ULP subsequently processes the entryand removes it from the CQ, freeing up a space in both the WQ and CQ.The number of entries available in the SQW are monitored by the ULP sothat it does not overwrite any valid entries. Likewise, the IPSOEmonitors the number of entries available in the CQ.

[0120]FIG. 11 is an exemplary diagram that depicts the contents of anentry in a Socket Context Table in accordance with the presentinvention. The Socket Context Table 1100 contains a Socket Context Entry(SCE) 1102 for each work queue pair (QP). These entries contain manyfields that are broken up into the Socket Context (SC), the Send WorkQueue Context (SWQC), Receive Work Queue Context (RWQC), Additional TCPContext, and IP Context.

[0121] The Socket Context includes Flags 1110, which contain the stateof the QP, the IP version being utilized, and the port number of the QP.The state of the QP is set by the IPSOE. The IP version and port numberof the QP are set by the Consumer. The Path Maximum Transfer Unit (PMTU)field 1112 contains the maximum data payload transfer size. The RetryFlags 1114 include the number of times a WQE is retried must be retriedby the IPSOE and the current number of retries that have been attempted.The latter is used by the IPSOE to keep track of the number of times theIPSOE has actually retried the WQE. The ACKT (Acknowledgement Timeout)field 1116 is the amount of time the IPSOE will wait for anacknowledgement (ACK) before marking the TCP Segment associated with theACK as lost.

[0122] The Window Size (WS) field 1118 contains the outbound TCP/IP andinbound TCP/IP window sizes, each in number of bytes, for theconnection. The Maximum Remote Direct Memory Access (MRDMA) field 1120is the maximum number of outstanding RDMA Read Requests from the remotesocket. The pending receive count (PRC) field 1122 is the number ofreceive packets that are pending acknowledgements (ACKs). The Queue PairType (QT) field 1124 describes the type of service associated with theQP (e.g. Sockets, iSCSI, DAFS, etc . . . ). The Data Segments (DS) field1126 is the maximum number of data segments per WQE. The protectiondomain (PD) field 1130 identifies the Protection Domain associated withthe ULP (in one embodiment, it is set to the process ID and is used toensure that the current operations have authority to access the memoryregion being read).

[0123] The send and receive work queue contexts contain similar dataexcept that they point to different queues. The Receive Work Queue HeadPointer Physical Address field 1146 points to the head of the circularwork queue for receive operations. The ULP writes to the head of the RWQwhen a send is initiated. The Receive CQ Index field 1148 is the indexinto the send completion queue for the associated QP. The Pending RQWECount 1150 is the number of pending RWQE's in the RWQ. The PendingReceive Xfer Count field 1152 is the number of pending receiveoperations. The Next RWQE field 1154 is a pointer to the next RWQE inthis RWQ to be processed. Each RWQE contains a list of Data Segments.For RDMA, each of the Data Segments contains a Steering Tag (STag),Virtual Address and Length. The IPSOE uses the STag to translate theVirtual Address into a Physical Address. The Logical Next Send TransferAddress 1142 and Physical Next Send Transfer Address 1144 are thevirtual and physical addresses (respectively) immediately following thelast outbound transfer as translated via the IPSOE's memory translationand protection table.

[0124] The send work queue context has fields that are analogous tothose in the receive work queue context, as represented in FIG. 11 aselements 1132-1165.

[0125] The TCP Context 1160 and IP Context 1162 contain informationregarding the type of TCP and IP connections that are being utilized astransport mechanisms.

[0126] When a receive operation is initiated, the ULP creates one ormore RQWE's 1202 that are written to the RWQ pointed to by the RWQ HeadPointer 1146, as shown in FIG. 12. The RWQ 1204 is a linked list ofpages of RWQEs. The last entry 1206 in each page is a pointer containingthe physical address to the next page of RWQEs.

[0127] When the RWQ is created, the initial RQW free space is set in avariable maintained by the ULP, referred to as the “RWQ Credit Count”.An RWQ Credit corresponds to one RQWE. As long as there is space in theRWQ (i.e. RQW Credit Count is non-zero), RWQE's can be added to the headof the list pointed to by the RWQ Head Pointer 1146. After some numberof RWQE's have been added to the RWQ, the ULP notifies the IPSOE that ithas done so via an “RWQ doorbell”, and then clears the RQW Credit Count.In a preferred embodiment, a doorbell is a memory mapped I/O operation(MMIO). The number of RWQE's added to the RWQ are indicated to the IPSOEin the RWQ doorbell. The IPSOE adds this count to the Pending RQWE Count1150, to track the number of pending RWQE's in the RWQ.

[0128] Returning to FIG. 12, an enlarged view of the RWQE 1202 is shownin the box having fields 1210-1222. As shown in FIG. 12, the RWQE 1202includes a Work Request ID field 1210 which is an identifier that isused to associate WQE's with eventual CQE's. The Op Type field 1212 isthe operation type. Operation types include: Send, Send with SolicitedEvent, RDMA Write, RDMA Read, or a Memory (e.g. Bind Memory Window)Operation. The Flags 1214 include information such as: SignaledCompletion requested; Immediate Data present; and Fence requested. Ifthe ULP requested Signaled Completion, then a Work Completion (WC) willbe returned by the IPSOE when the SWQE completes. If Immediate Data isrequested, then the SWQE contains data that the IPSOE must send asImmediate Data on the outbound transfer. Finally, if the RWQE contains aFence, then the IPSOE must wait for the RWQE to complete, beforeprocessing the next RWQE. The Fence operation can be used for Memory andRDMA Read Operations.

[0129] The Number of Data Segments field 1216 is the quantity of DataSegments that are to be transmitted. Each Data Segment 1218-1222contains a STag, Virtual Address, and Length. The IPSOE's MemoryTranslation and Protection Table uses these 3 fields to access the datareferenced by the data segment. In an iSCSI environment this list ofaddresses is replaced by a pointer to the iSCSI command which willcontain the destination IP address and port number along with a list ofphysical addresses of data that is to be transmitted.

[0130] After the data pointed to by the list of addresses in the RWQEsis transferred by the IPSOE to host memory, the IPSOE must notify theconsumer that the work is completed. This is carried out through the useof a completion queue (CQ). CQ's are created by the ULP for each IPSOE.At the time of creation, the size of the CQ is set (i.e. number of CQE'sthe CQ can hold). FIG. 13 shows the CQ context along with the detail ofthe CQE. The Socket Completion Queue Context Table 1300 containsCompletion Context Entries (CCE) such as 1302 associated with a givenWQ. Each CCE 1302 contains a CQ Tail Pointer Physical Address field 1304which is a pointer to the address of an entry such as 1322 in thecircular linked list, which is the CQ 1320. This is a linked list of CQEpages where the last entry in each page is a pointer to the next page ofthe list.

[0131] Each CQE contains a pointer to the send operation that completed.This is accomplished by utilizing a WQ Number field 1330 and a WQE indexfield 1332 that point to the WQE that has completed. The Send/Receivefield 1334 of the CQE identifies the type of operation (Send WQE orReceive WQE) that completed and the Completion Status field 1336contains information as to whether the WQE completed successfully orunsuccessfully.

[0132] The WQE that is pointed to is shown in 1340-1352. For a CQE thatis associated with a Send WQE, all the fields contained in the Send WQEare contained in the CQE. Similarly, for a CQE that is associated with aReceive WQE, all the fields contained in the Receive WQE are containedin the CQE. The Work Request ID field 1340 is a 64-bit identifier thatis used to associate this completion queue entry back to a specific WQE.

[0133] As operations complete, the IPSOE writes to the tail of the CQusing the CQ Tail Pointer 1304, assuming the CQE Free Space count 1308in the Socket CQ Context 1300 indicates that there is available freespace. CQE Free Space is a count of the number of free entries in theCQ. If no space is available CQ (i.e. CQE Free Space is zero) then aninterrupt will be issued up to the ULP. At that point the ULP mayincrease the size of the CQ and notify the IPSOE how many entries havebeen added to the CQ. The IPSOE responds by adjusting the CQE Free Spacecount up by the indicated amount. The ULP keeps track of the last CQEthat it read and when the next entry becomes valid. The ULP ensures thatthe operation completed successfully and removes CQE from the head ofthe CQ by invalidating the entry and advancing a software CQ headpointer that it maintains. The ULP also maintains counts of the numberof CQE's it has removed from the CQ, both in total, and on a per WQbasis. The per WQ counts are maintained in “RWQ Credit Count” variables1600, as shown in FIG. 16, managed by ULP software. RWQ Credit Countconservatively indicates to the ULP how much free space there is in thecorresponding RWQ. When the ULP issues an RWQ Doorbell 1602 to theIPSOE, it passes the corresponding RWQ Credit Count to the IPSOE in thedoorbell as an RWQE Count, and then clears the RWQ Credit Count.

[0134] The total number of CQE's the ULP removes from a CQ aremaintained in a “CQ Credit Count” variable 1700, as shown in FIG. 17,managed by the ULP in software. The ULP indicates to the IPSOE how manyCQE's it has removed from a CQ by passing the CQ Credit Count to theIPSOE in a “CQ Doorbell” 1702. A CQ doorbell is an MMIO like the RWQDoorbell. However, instead of indicating how many RWQE's have been addedto a RWQ, a CQ doorbell indicates how many CQE's the ULP has removedfrom a specified CQ. When the ULP issues a CQ Doorbell to the IPSOE, itpasses the CQ Credit Count to the IPSOE in the doorbell, and then clearsthe CQ Credit Count. The IPSO adds the CQ Credits to the CQ Free Spacecount 1308 of the Socket CQ Context 1300 (that is also specified in theCQ doorbell). When the IPSOE adds one or more CQE's to the tail of a CQ1703, it decreases the CQ Free Space Count by that amount. Hence the CQFree Space count conservatively indicates to the IPSOE the amount offree space in the corresponding CQ.

[0135] Note the implicit flow of RWQ and CQ Credits between the ULP andthe IPSOE. As the ULP consumes CQEs from a CQ, it acquires CQ and RWQCredits. As the ULP issues RWQ Doorbells to the IPSOE it consumes RWQCredits, and implicitly passes them to the IPSOE. The IPSOE in turnimplicitly returns RWQ Credits to the ULP as it posts CQE's in a CQ.Conversely, the IPSOE implicitly consumes CQ Credits as it posts CQE'sto a CQ. The ULP explicitly returns CQ Credits to the IPSOE in CQDoorbells.

[0136] Now turning to FIG. 14 which is a flowchart that outlines anexemplary process for creating a queue pair. Step 1410 starts theflowchart. The consumer initiates the creation of queue pair by callingthe ULP to create a queue pair (step 1400). 1402 the ULP then allocatesand pins memory for the queue pair context or socket context as depictedin FIGS. 11 and 12 (Step 1402). Once the QP is created including thesetting attributes such as the number of WQEs allowed in the QP, thencontrol is returned to the consumer (step 1404) and the process ends(step 1406).

[0137]FIG. 15 is a flowchart outlining an exemplary operation of thepresent invention for send transactions. As shown in FIG. 15 theconsumer creates a number of send work requests 1500 and hands them offto the ULP 1502. The ULP converts the work requests into SWQE's 1504 asdepicted in FIG. 12. The ULP writes the RWQE's 1202, to the addresspointed to by the Receive WQ Head Pointer Physical Address 1132 and thenupdates this pointer 1506. The ULP notifies the IPSOE via a RWQ Doorbellhow many RWQE's were posted to the RWQ, and clears the associated RWQCredit Count.

[0138] The RWQE is processed by the IPSOE hardware by transmitting thedata in the data segments pointed to by the Data Segment Addresses1218-1222 within the RWQE 1202 1510. Once the entire RWQE has beenprocessed, then the IPSOE creates a CQE 1512, and decrements the PendingRWQE Count. The IPSOE writes the CQE into the CQ at the address pointedto by the CQ Tail Pointer Physical Address 1304, updates this pointer tothe next available CQE 1514, and decrements the CQE Free Space count.The IPSOE then notifies the ULP of a new CQE 1516. This can be done byseveral methods with one implementation being an interrupt pollingmechanism between the IPSOE and the ULP. Once notified, the ULPprocesses the CQE's it removes from the CQ and updates the correspondingCQ and RWQ Credit counts 1518. At this point the entire send operationhas ended 1520.

[0139]FIGS. 18A and 18B illustrate a diagram and flowchart,respectively, depicting the initialization of the TCP SelectiveAcknowledgement (SACK) Table in accordance with the present invention.The SACK Table Initialization flowchart 1800 is performed when theConsumer, such as Consumer 1840 invokes a Modify QP. As shown in FIG.18B, the Consumer issues a Modify QP to the IPSOE (step 1850), such asIPSOE 1828. The IPSOE determines if Modify QP input modifiers haveselected SACK support to be enabled on the QP (step 1854). If SACKsupport is not enabled on the QP, the IPSOE exits the flowchart. If SACKsupport is enabled, the IPSOE initializes the SACK table (step 1858),such as SACK Pane Table 1808 in IPSOE 1828. The IPSOE sets the number ofentries in the SACK table to the number of SACK entries selected by theModify QP verb.

[0140] The first entry (i.e. the ACK'd SN entry 1812) in the tablecontains these fields: the last Acknowledged TCP Sequence Number; thecurrent RQ, such as RQ 1824, WQE, and the offset into the WQE. Eachsubsequent entry in the table, such as Pane Entry 1816 and 1820,represents a Pane Entry and contains these fields: the starting SN (TCPSequence Number) represents the first byte successfully received andSACK'd, the ending SN represents the last byte successfully received andSACK'd, the current WQE associated with the ending SN, and the offsetinto the WQE associated with the ending SN. All these subsequent entriesare initialized to zero. At this point the IPSOE exits the flowchart.

[0141]FIG. 19 is a flowchart outlining the Selective Acknowledgementprocessing performed by the IPSOE when an incoming TCP/IP Segment isprocessed in accordance with the present invention. The SACK IncomingPacket Processing flowchart 1900 is performed when an incoming packetarrives that is associated with an IPSOE QP that has SACK enabled (step1902). The IPSOE determines if the incoming packet contains a TCPSequence Number (SN) larger than next expected, but within the TCPWindow (step 1904). If not, the packet is dropped and the operationends. If so, the IPSOE compares the starting SN-1 of the incoming packetto the ACK'd SN in the Pane Table Entry (PTE) (step 1908).

[0142] If the starting SN-1 of the incoming packet is equal to the ACK'dSN in the Pane Table Entry (PTE), the IPSOE: inserts the SNcorresponding to the last byte of the packet in the last ACK'd SN PTEand places the data from the packet into the current WQE and WQE offsetstored in the PTE (step 1912). The IPSOE then checks if the SN stored inthe ACK'd SN PTE overlaps with the first Pane entry (step 1916), theoperation continues to step 1926 where the IPSOE: collapses the twopanes and places the data from the packet into the current WQE and WQEoffset associated with the first byte of the incoming packet's SN (step1926).

[0143] If the SN-1 of the incoming packet is not equal to the ACK'd SNin the PTE (step 1908), the IPSOE compares the starting SN-1 of theincoming packet to the Pane Entries in the Pane Table Entry (PTE) anddetermines if SN-1 is within the SNs of an existing Pane Table Entry(step 1924). If it is not within the SNs of an existing Pane Entry, theoperation continues to step 1925. If it is within the SNs of an existingPane Table Entry, the operation continues to step 1926 where the IPSOE:collapses the two panes and places the data from the packet into thecurrent WQE and WQE offset associated with the first byte of theincoming packet's SN (step 1926).

[0144] If no empty PTEs are available (step 1925), the packet is droppedand the operation terminates. If an empty PTE is available, the IPSOE:creates the PTE associated with the incoming packet and places the datafrom the packet into the current WQE and WQE offset associated with thefirst byte of the incoming packet's SN (step 1928). After either theexecution of steps 1926 or 1928, the operation continues to step 1930.

[0145] If the SACK table was updated (step 1930), the IPSOE sends a SACKcontaining the contents of the SACK table to the source of the incomingpacket (step 1932) and the operation terminates. Otherwise, theoperation simply terminates.

[0146] The remaining figures in this patent describe the IPSOEvirtualization mechanisms. The basic philosophy used by these mechanismsconsists of the following: During IPSOE Driver and Library development,the IPSOE Driver and Library are each segmented into: an IPSOE HostedServer Driver, which runs in the Hosted Server; an IPSOE Hosted ServerLibrary, which runs in the Hosted Server; and an IPSOE Hosting PartitionDriver, which runs in the Hosting Partition (HP). A Software Queue Pairis provided between IPSOE Hosted Partition and the IPSOE HostingPartition (HP). A single physical server is partitioned into multiplevirtual servers. Each Hosted Server is a virtual server running anoperating system instance on a single physical server. The HostingPartition is also one of these virtual servers that includes themechanism described in this docket.

[0147] The HP uses an IPSOE Resource Management Table (RMT) to assignphysical IPSOE resources to the virtual Hosted Servers. Each entry ofthe IPSOE RMT contains the resources assigned to the HS associated withthe entry. After the entry is made, the HS can access the IPSOE and usethe IPSOE resources allocated in the RMT. Through the use of theSoftware Queue Pair and the Resource Management Table the HP allows asingle IPSOE to be shared between multiple Hosted Servers.

[0148] The IPSOE Hosted Partition Driver/Library places IPSOE operationson the IPSOE Software Send Queue. The operations consist of all standardand vendor unique IPSOE verbs. All IPSOE operations performed by theHosted Server's (HS) Operating System are placed on the IPSOE SoftwareSend Queue and, if necessary, invoke the Hosting Partition through atrap call.

[0149] The IPSOE Hosting Partition Driver performs operations posted tothe IPSOE Software Send Queue and returns results to the HostedPartition through the IPSOE Software Receive Queue. The HostingPartition performs all the operations, including interrupt processing.

[0150] The IPSOE uses a Server Domain (SD) field to associate IPSOE QP,CQ, and Memory Translation and Protection Table (TPT) Resources to avirtual Hosted Server (HS). For each HS running on the physical server,the HP allocates an SD on the IPSOE (see FIG. 22). For each IPSOE QPResource used by the HS, the HP assigns a SD and MAC (Media AccessControl) Address Table Entry (MAC Table Entry) to the IPSOE QP (see FIG.27). For each IPSOE CQ Resource used by the HS, the HP assigns a SD tothe IPSOE CQ (see FIG. 26) Resource, respectively. Similarly, for eachIPSOE Memory Resource used by the HS, the HP assigns a SD to the IPSOEMemory (see FIG. 28) Resource, respectively.

[0151] When the IPSOE performs an operation between a QP and a CQ orMemory TPT Resource, the IPSOE must verify that the SD value matches.When an incoming TCP Segment arrives, the IPSOE must verify that theincoming TCP Segment's destination MCA Address and Port match the MCAAddress and Port assigned to the QP referenced by the TCP Quintuple(i.e. the Transport Type, Source Port Number, Destination Port Number,Source IP Address, and Destination IP Address). FIGS. 26, 27, and 28describe the flowcharts used to associate a SD to a CQ, QP, and MemoryTPT, respectively.

[0152] For any type of incoming RDMA Protocol Send message, if MAC TableEntry matches, the incoming TCP Segment is indeed associated with theHosted Server that is assigned to the QP and TCP Segment processingcontinues. Otherwise TCP Segment processing is terminated. FIG. 32describes the flowchart used to perform the MAC Address and Port Numberverification.

[0153] For any type of incoming RDMA Protocol Write or Read message, theServer Domain (SD) stored in the Memory TPT must match the SD stored inthe QP that is associated with the incoming TCP Segment and the MACTable Entry stored in the QP must match the MAC Address and Port Numberreferenced in the incoming TCP Segment. If both match, the processingcontinues on the incoming TCP Segment. Otherwise it is terminated. FIG.32 describes the flowchart used to perform the SD verification.

[0154] All IPSOE verb invocations by the Hosted Server are passedthrough the Hosting Partition. The Hosting Partition performs the actualIPSOE verbs through the necessary programmed I/O operations. FIGS. 24through 31 describe the flowcharts used: to pass IPSOE verb invocationsfrom the HS to the HP, by the HP to perform the verb invocation, andreturn the verb results back to the HS. Some IPSOE verb invocationsrequire Direct Memory Access (e.g. Post Send, Post Receive, and incomingRDMAs). For verbs that required Direct Memory Access (DMA), the HostingPartition sets up the DMAs as Redirected DMAs that directly target theMemory Addresses. When the Redirected DMA is no longer needed, the IPSOEdestroys it. FIGS. 30-31 describe the set up and destruction ofRedirected DMAs.

[0155]FIG. 20 is an exemplary diagram of the operations supported by theIPSOE and the tables used to maintain state for those operations inaccordance with the present invention. All verb invocations by aconsumer process running in a Hosted Server's (HS) Operating System,such as HS Consumer 2100, must go through the Hosting Partition (HP),such as HP 2104. The Hosting Partition maintains an IPSOE ResourceMapping Table (RMT), such as IPSOE Resource Mapping Table 2150, whichcontains the maximum resources available to the HS and the actualresources that have already been allocated to the HS. The IPSOE, such asIPSOE 2190, supports: a set of Physical IPSOE Management Verbs (Open,Query, Modify, Close) 2108; a Set Asynchronous Event Handler Verb 2112;a set of Server Domain (SD) Management Verbs (Allocate, Deallocate)2116; a set of IP Address Alias (IP Alias) Verbs (Allocate, Deallocate)2117; a set of CQ Management Verbs (Create, Poll, and Set CQ Handler)2128; a set of Memory Region Management Verbs (various Register verbs,Allocate, various Deregister verbs, various Reregister verbs) 2136; aset of QP Management Verbs (Create, Modify, Query, Destroy, Post Send,and Post Receive) 2120; a set of IPSOE Direct Memory Access (DMA)Operations (Write and Read) 2144; a Port MAC Table 2110; a SD ContextTable 2118; an IP Address Alias (IP Alias) Table 2119; a CQ ContextTable 2132; a Memory Translation and Protection Table (TPT) 2140; and QPContext Table 2124. All the verbs are familiar to those experienced inRDMA technology (Virtual Interface Architecture standard, InfiniBandstandard, and, more recently, RDMA over IP standard), except for the SDManagement and the IP Alias Verbs. The SD Management Verbs are used toallocate and deallocate Server Domain entries from the IPSOE's SDContext Table. Similarly, the IP Alias Verbs are used to allocate anddeallocate IP Alias entries from the IPSOE's IP Alias Table.

[0156] We now turn to, FIG. 21 which is an exemplary diagram of theIPSOE MAC Table and the Hosting Partition Resource Mapping Table, whichis used by the Hosting Partition to map between physical IPSOE Resourcesand Virtual IPSOE Resources, in accordance with the present invention.This figure depicts an IPSOE Resource Mapping Table (RMT), such as IPSOEResource Mapping Table 2112. An IPSOE RMT is maintained in the HostingPartition, such as HP 2104, for every IPSOE, such as IPSOE 2100,accessed through the HP. The Server Identifier (Server ID), such asServer ID 2108, of the Hosted Server is used to access the IPSOEResources allocated to the HS through the HP.

[0157] Each row in the table contains three types of entries: VariableResource 2190 entries, Consumer Assigned Resource 2194 entries, andFixed Resource 2198 entries. Each Variable Resource 2190 entry typecontains a Maximum and an Actual value. The Maximum value reflects thenumber of physical IPSOE Resources that have been allocated to the HSwith a Server ID that is associated to the row. The Actual valuereflects the number of physical IPSOE Resources that are currently inuse by the HS with Server ID that is associated to the row. Variableresources include Number of QPs (2116 and 2120), Number of WQEs per QP(2124 and 2128), Number of CQs (2132 and 2136), Number of MemoryRegions, Number of Windows, and other variable IPSOE resources.

[0158] Each Consumer Assigned Resource 2194 contains a single field thatreflects the value for the resource that is associated to the HS with aServer ID that is associated to the row. Consumer Assigned Resourcesinclude the Server Domain (2148) assigned to the Server ID and the IPSOEMAC Table Index (2144) assigned to the Server ID. The IPSOE MAC TableIndex 2144 and IPSOE Port Number 2156 are used to index into the IPSOE'sPort Mac Table.

[0159] Finally, each Fixed Resource 2198 contains a single field thatreflects the value for the resource that is associated to the HS with aServer ID that is associated to the row. For example, the IPSOE PortNumber 2156 is one such Resource. The first row is a physical IPSOE rowwhich reflects the maximum number of physical IPSOE resources and thetotal number that are currently in use across all HSs and the HP. Forthe first row: each of the Maximum Variable Resource entries reflectsthe maximum number of physical resources the IPSOE supports for thatvariable resource; each of the Actual Variable Resource entries reflectsthe total number of physical ISPOE resources that have already beenallocated through the HP; and each of the Fixed Resource entriesreflects the total number of physical resources the IPSOE supports forthat fixed resource. The Consumer Assigned Resources are not used forthe first row. The second row is the HPs row and reflects the IPSOEresources allocated to the HP and already used by the HP.

[0160]FIG. 22 is a flowchart outlining the IPSOE VirtualizationInitialization mechanism in accordance with the present invention. Step1 2200: Hypervisor consumer queries I/O adapter to determine if it is anIPSOE (e.g. Open IPSOE). If it is an IPSOE, it continues to Step 2.Otherwise it is not an IPSOE and the operation terminates.

[0161] Step 2 2204: Hypervisor requests Hosting Partition to load adevice driver for the IPSOE (using, for example, the mechanism describedin co-pending and commonly assigned U.S. patent application Ser. No.10/132,461 (Attorney Docket No. AUS920020065US1) and passes the IPSOEHandle to the Hosting Partition Device Driver (HP).

[0162] Step 3 2208: HP performs Query IPSOE to determine if IPSOEsupports IP address and MAC address resource virtualization. If it does,HP continues to step 4: 2212. Otherwise IPSOE IP address and MAC addressresources are not virtualized and the operation ends.

[0163] Step 4 2212: HP passes Server Manager (SM could be a human beingor a program, such as a Virtual Server Partition Manager that also runsin the Hosting Partition) the number of Server Domains the IPSOEsupports, the number of MAC Addresses IPSOE supports, and a tablecontaining all the IPSOE Resources the IPSOE supports.

[0164] Step 5 2216: Server Manager creates the IPSOE Resource MappingTable: The Hosted Server's ID is used an index into the table. For eachvariable IPSOE Resource, the table contains two entries: one reflectingthe maximum number allocated to the HS; and the second the actual numberin use by the HS. For each consumer assigned IPSOE Resource, the tablecontains the value assigned by the SM. For each fixed IPSOE Resource,the table contains the value of the fixed Resource that was returned bythe Query IPSOE verb.

[0165] Step 6 2220: The Server Manager passes the IPSOE Resource MappingTable to the HP, along with the number of entries in the table.

[0166] Step 7 2224: The HP: stores the IPSOE Resource Mapping Table;initializes the Variable Resource entries of the first row by setting,for each Variable Resource, the first entry to the maximum number ofresources the IPSOE supports for that variable resource and the secondentry to the total number of resources that have already been allocated(which initially is just the resources allocated to the HP); andinitializes the second row with the number of resources the HP willrequire for its own use.

[0167] Step 8 2228: The HP issues the Modify IPSOE to the IPSOE, settingthe number of Server Domains (SDs) the IPSOE will support to the size ofthe IPSOE Resource Mapping Table.

[0168]FIG. 23 is a flowchart outlining the Hosted Server Creationmechanism in accordance with the present invention. Step 1 2300: ServerManager requests the HP to create a Hosted Server entry by passing theMaximum Number of each IPSOE Resource that is to be assigned to theHosted Server in a field called Requested_HS_Resources.

[0169] Step 2 2304: HP checks to see if IPSOE Resource Mapping Table hasan empty entry available for use. If an entry is empty, the HP continuesto step 3. Otherwise it returns to the SM with an out of IPSOE RMTentries error (2306).

[0170] Step 3 2308: For each Variable IPSOE Resource in the IPSOE RMT,the HP adds the number of the Variable IPSOE Resource in the first rowof the RMT to the associated Variable Resource contained in theRequested_HS_Resources field and stores the result into theAllocated_Variable_IPSOE_Resources field.

[0171] Step 4 2312: For each Variable IPSOE Resource in the IPSOE RMT,the HP compares the maximum number of the Variable IPSOE Resource in thefirst row of the RMT to the associated maximum number in theAllocated_Variable_IPSOE_Resources.

[0172] Step 4.1 2316: If any maximum Variable IPSOE Resource entry inthe first row is less than the associated entry in theAllocated_Variable_IPSOE_Resource, the HP creates an out of IPSOEResources error and exits the HS Creation Flowchart by passing controlback to the SM (2318).

[0173] Step 4.2 2320: If each maximum Variable IPSOE Resource entry inthe first row is greater than the associated entry in theAllocated_Variable_IPSOE_Resource, the HP allocates an IPSOE ResourceMapping Table entry for the Hosted Server and stores into it theRequested_HS_Resources.

[0174] Step 5 2324: HP returns successful completion back to the SM.

[0175]FIG. 24 is a flowchart outlining the Hosted Server Operation Trapsto Hosting Partition mechanism in accordance with the present invention.Step 1 2400: Hosted Server invokes IPSOE operation by creating the verbinput modifiers and passing the verb to the Hosting Partition throughthe HS_SSQ call.

[0176] Step 2 2404: Hosting Partition reads entries in the HS_SSQ inFirst-In, First-Out (FIFO) order.

[0177] Step 3 2408: The Hosting Partition invokes the verb functionissued by the Hosted Server:

[0178] Step 3.1 2412: If the verb is an IPSOE Management Verb, HPinvokes the IPSOE Management flowchart on FIG. 25 (2414).

[0179] Step 3.2 2416: If the verb is a CQ Management Verb, HP invokesthe CQ Management flowchart on FIG. 26 (2418).

[0180] Step 3.3 2420: If the verb is an QP Management Management Verb,HP invokes the CQ Management flowchart on FIG. 27 (2422).

[0181] Step 3.4 2424: If the verb is an Memory Management Verb, HPinvokes the Memory Management flowchart on FIG. 28 (2426).

[0182] Step 3.5 2428: If the verb is an PD and IP Alias Management Verb,HP invokes the PD and IP Alias Management flowchart on FIG. 29 (2430).

[0183] Step 3.6 2432: If the verb is an Post Send and Post Receive Verb,HP invokes the Post Send and Post Receive flowchart on FIG. 30 (2434).

[0184] Step 3.7 2436: If the verb is an Poll CQ, Set CQ Handler, SetAsync Event Handler Verb, HP invokes the Poll CQ, Set CQ Handler, SetAsync Event Handler flowchart on FIG. 31.

[0185] Step 4 2456: The Hosting Partition returns the results of theverb function through the HS_SRQ.

[0186]FIG. 25 is a flowchart outlining the IPSOE Management Verbs (Open,Query, Modify, and Close) mechanism in accordance with the presentinvention. Step 1 2500: The Hosting Partition (HP) uses the HostedServer's (HS) Server ID to index into the IPSOE Resource Mapping Table.

[0187] Step 2 2504: If the verb is an Open IPSOE, the HP checks theIPSOE Resource Mapping Table entry that is associated with Hosted Server(HS) and the first entry in the IPSOE Resource Mapping Table.

[0188] Step 2.1 2508: If the entry in the IPSOE Resource Mapping Tablealready contains an IPSOE Handle, an error result is created and passedback to the HP verb mapping mechanism (FIG. 24) (2510).

[0189] Step 2.2 2512: If the first entry in the IPSOE Resource MappingTable contains an IPSOE Handle and the entry in the IPSOE ResourceMapping Table does not contain an IPSOE Handle, the HP writes the IPSOEHandle from the first entry in the IPSOE Resource Mapping Table to theentry in the IPSOE Resource Mapping Table that is associated with theHS. The HP passes back a good result, along with the IPSOE verb outputmodifiers that are associated with the HS, to the HP verb mappingmechanism (FIG. 24) (2514).

[0190] Step 2.3 2516: If the first entry in the IPSOE Resource MappingTable does not contain an IPSOE Handle, the HP an error result iscreated and passed back to the HP verb mapping mechanism (FIG. 24).

[0191] Step 3 2520: If the verb is a Query IPSOE verb, the HP checks ifthe entry in the IPSOE Resource Mapping Table that is associated withHosted Server (HS) contains an IPSOE Handle (2524).

[0192] Step 3.1 2524: If the entry in the IPSOE RMT contains an IPSOEHandle, the HP creates a Query IPSOE verb and passes it to the IPSOE.For each Variable IPSOE Resource, the HP passes back the maximum valuestored in the IPSOE RMT entry, along with all fixed IPSOE Resourcesreturned from the Query IPSOE verb, to the HP verb mapping mechanism(FIG. 24) (2526).

[0193] Step 3.2 2528: If the entry in the IPSOE Resource Mapping Tabledoes not contain an IPSOE Handle, an error result is created and passedback to the HP verb mapping mechanism (FIG. 24) (2528).

[0194] Step 4 2532: If the verb is a Modify IPSOE verb, the HP checks ifthe entry in the IPSOE Resource Mapping Table that is associated withHosted Server (HS) contains an IPSOE Handle (2536).

[0195] Step 4.1 2536: If the entry in the IPSOE Resource Mapping Tablecontains an IPSOE Handle, the HP creates a Modify IPSOE verb and passesonly fields that can be modified by the HS to the IPSOE. The HP passesback the result of the Modify IPSOE verb, along with the IPSOE verboutput modifiers that are associated with the HS, to the HP verb mappingmechanism (FIG. 24) (2538).

[0196] Step 4.2 2540: If the entry in the IPSOE Resource Mapping Tabledoes not contain an IPSOE Handle, an error result is created and passedback to the HP verb mapping mechanism (FIG. 24).

[0197] Step 5 2544: If the verb is a Close IPSOE verb, the HP checks ifthe entry in the IPSOE Resource Mapping Table that is associated withHosted Server (HS) contains an IPSOE Handle.

[0198] Step 5.1 2548: If the entry contains an IPSOE Handle, the HP:clears the IPSOE handle from the IPSOE RMT entry; and passes back a goodresult, along with the IPSOE verb output modifiers that are associatedwith the HS, to the HP verb mapping mechanism (FIG. 24).

[0199] Step 5.2 2552: If the entry does not contain an IPSOE Handle, anerror result is created and passed back to the HP verb mapping mechanism(FIG. 24).

[0200]FIG. 26 is a flowchart outlining the CQ Management Verbs (Create,Query, Modify, and Destroy) mechanism in accordance with the presentinvention. Step 1 2600: If the verb is a Create CQ, the HP retrieves theMaximum Number of CQs and Actual Number of CQs from the IPSOE ResourceMapping Table entry that is associated with Hosted Server (HS) (2602).

[0201] Step 1.1 2604: If Maximum Number of CQs=Actual Number of CQs, nomore CQs are available for the HS and an error result is created andpassed back to the HP verb mapping mechanism (FIG. 24) (2606).

[0202] Step 1.2 2608: If Maximum Number of CQs<Actual Number of CQs, theHP: increases the Actual Number of CQs by 1 and stores the results inthe Actual Number of CQs cell of the IPSOE Resource Mapping Table entrythat is associated with Hosted Server (HS); builds a Create CQ verbusing the input modifiers passed by the HS and the SD associated withthe HS (from the IPSOE RMT entry that is associated with the HS); passesthe Create CQ verb to the IPSOE; and passes back the Create CQ IPSOEverb result, along with the IPSOE verb output modifiers that areassociated with the HS, to the HP verb mapping mechanism (FIG. 24).

[0203] Step 2 2612: If the verb is a Query CQ verb, the HP: builds theQuery CQ verb; passes the Query CQ verb to the IPSOE; and passes backthe Query CQ verb result from the IPSOE, along with the IPSOE verboutput modifiers that are associated with the HS, to the HP verb mappingmechanism (FIG. 24) (2614).

[0204] Step 3 2616: If the verb is a Modify CQ verb, the HP: builds theModify CQ verb; passes the Modify CQ verb to the IPSOE; and passes backthe Modify CQ verb result from the IPSOE, along with the IPSOE verboutput modifiers that are associated with the HS, to the HP verb mappingmechanism (FIG. 24)(2618).

[0205] Step 4 2620: If the verb is a Destroy CQ verb, the HP: builds theDestroy CQ verb; passes the Destroy CQ verb to the IPSOE; and passesback the Destroy CQ verb result from the IPSOE, along with the IPSOEverb output modifiers that are associated with the HS, to the HP verbmapping mechanism (FIG. 24).

[0206]FIG. 27 is a flowchart outlining the QP Management Verbs (Create,Query, Modify, and Destroy) mechanism in accordance with the presentinvention. Step 1 2700: If the verb is a Create QP, the HP retrieves theMaximum Number of QPs and Actual Number of QPs from the IPSOE ResourceMapping Table entry that is associated with Hosted Server (HS) (2702).

[0207] Step 1.1 2704: If Maximum Number of QPs=Actual Number of QPs, nomore QPs are available for the HS and an error result is created andpassed back to the HP verb mapping mechanism (FIG. 24) (2706).

[0208] Step 1.2 2708: If Maximum Number of QPs<Actual Number of QPs, theHP: increases the Actual Number of QPs by 1 and stores the results inthe Actual Number of QPs cell of the IPSOE Resource Mapping Table entrythat is associated with Hosted Server (HS); builds a Create QP verbusing the input modifiers passed by the HS, the IP Virtual Address, theSD associated with the HS (from the IPSOE RMT entry that is associatedwith the HS), and the MAC Address associated with the HS (from the IPSOERMT entry that is associated with the HS); passes the Create QP verb tothe IPSOE; and passes back the Create QP IPSOE verb result, along withthe IPSOE verb output modifiers that are associated with the HS, to theHP verb mapping mechanism (FIG. 24) (2708).

[0209] Step 2 2712: If the verb is a Query QP verb, the HP: builds theQuery QP verb; passes the Query QP verb to the IPSOE; and passes backthe Query QP verb result from the IPSOE, along with the IPSOE verboutput modifiers that are associated with the HS, to the HP verb mappingmechanism (FIG. 24) (2714).

[0210] Step 3 2716: If the verb is a Modify QP verb, the HP: builds theModify QP verb; passes the Modify QP verb to the IPSOE; and passes backthe Modify QP verb result from the IPSOE, along with the IPSOE verboutput modifiers that are associated with the HS, to the HP verb mappingmechanism (FIG. 24) (2718).

[0211] Step 4 2720: If the verb is a Destroy QP verb, the HP: builds theDestroy QP verb; passes the Destroy QP verb to the IPSOE; and passesback the Destroy QP verb result from the IPSOE, along with the IPSOEverb output modifiers that are associated with the HS, to the HP verbmapping mechanism (FIG. 24).

[0212]FIG. 28 is a flowchart outlining the Memory Management Verbs (AllRegisters, All Reregisters, All Allocates, All Deregisters) mechanism inaccordance with the present invention. Step 1 2800: If the verb is anytype of Register verb, the HP retrieves the Maximum Number of MemoryRegions and Actual Number of Memory Regions (MR) from the IPSOE ResourceMapping Table entry that is associated with Hosted Server (HS) (2802).

[0213] Step 1.1 2804: If Maximum Number of Memory Regions=Actual Numberof Memory Regions, no more MRs are available for the HS and an errorresult is created and passed back to the HP verb mapping mechanism (FIG.24) (2806).

[0214] Step 1.2 2808: If Maximum Number of MRs<Actual Number of MRs, theHP: increases the Actual Number of MRs by 1 and stores the results inthe Actual Number of MRs cell of the IPSOE Resource Mapping Table entrythat is associated with HS; builds a Register MR verb using the inputmodifiers passed by the HS and the SD associated with the HS (from theIPSOE RMT entry that is associated with the HS); passes the Register MRverb to the IPSOE; and passes back the Register MR verb result from theIPSOE, along with the IPSOE verb output modifiers that are associatedwith the HS, to the HP verb mapping mechanism (FIG. 24).

[0215] Step 2 2812: If the verb is any type of Reregister verb, the HP:builds a Reregister MR verb using the input modifiers passed by the HSand the SD associated with the HS (from the IPSOE RMT entry that isassociated with the HS); passes the Reregister MR verb to the IPSOE; andpasses back the Reregister MR verb result from the IPSOE, along with theIPSOE verb output modifiers that are associated with the HS, to the HPverb mapping mechanism (FIG. 24) (2814).

[0216] Step 3 2816: If the verb is any type of Allocate verb, the HP:increases the Actual Number of MRs by 1 and stores the results in theActual Number of MRs cell of the IPSOE Resource Mapping Table entry thatis associated with Hosted Server (HS); builds an Allocate MR verb usingthe input modifiers passed by the HS and the SD associated with the HS(from the IPSOE RMT entry that is associated with the HS); passes theAllocate MR verb to the IPSOE; and passes back the Allocate MR verbresult from the IPSOE, along with the IPSOE verb output modifiers thatare associated with the HS, to the HP verb mapping mechanism (FIG. 24)(2818).

[0217] Step 4 2820: If the verb is any type of Deregister verb, the HPbuilds a Deregister MR verb using the input modifiers passed by the HSand the SD associated with the HS (from the IPSOE RMT entry that isassociated with the HS); passes the Deregister MR verb to the IPSOE.

[0218] Step 4.1 2824: If the Deregister MR was successful, the HP:decreases the Actual Number of MRs by 1 and stores the results in theActual Number of MRs cell of the IPSOE Resource Mapping Table entry thatis associated with Hosted Server (HS); and passes back the Deregister MRverb result from the IPSOE, along with the IPSOE verb output modifiersthat are associated with the HS, to the HP verb mapping mechanism (FIG.24) (2826).

[0219] Step 4.2 2828: If the Deregister MR was unsuccessful, the HPpasses back the erroneous verb result from the IPSOE, along with theIPSOE verb output modifiers that are associated with the HS, to the HPverb mapping mechanism (FIG. 24).

[0220]FIG. 29 is a flowchart outlining the PD and IP Alias ManagementVerbs (Allocate PD, Deallocate PD) mechanism in accordance with thepresent invention. Step 1 2900: If the verb is an Allocate PD verb, theHP builds an Allocate PD verb using the input modifiers passed by theHS; passes the Allocate PD verb to the IPSOE; and passes back theAllocate PD verb result from the IPSOE, along with the IPSOE verb outputmodifiers that are associated with the HS, to the HP verb mappingmechanism (FIG. 24) (2902).

[0221] Step 2 2904: If the verb is an Deallocate PD verb, the HP buildsan Deallocate PD verb using the input modifiers passed by the HS; passesthe Deallocate PD verb to the IPSOE; and passes back the Deallocate PDverb result from the IPSOE, along with the IPSOE verb output modifiersthat are associated with the HS, to the HP verb mapping mechanism (FIG.24) (2906).

[0222] Step 3 2908: If the verb is an Allocate IP Alias verb, the HPbuilds an Allocate IP Alias verb using the input modifiers passed by theHS; passes the Allocate IP Alias verb to the IPSOE; and passes back theAllocate IP Alias verb result from the IPSOE, along with the IPSOE verboutput modifiers that are associated with the HS, to the HP verb mappingmechanism (FIG. 24) (2910).

[0223] Step 4 2912: If the verb is an Deallocate IP Alias verb, the HPbuilds an Deallocate IP Alias verb using the input modifiers passed bythe HS; passes the Deallocate IP Alias verb to the IPSOE; and passesback the Deallocate IP Alias verb result from the IPSOE, along with theIPSOE verb output modifiers that are associated with the HS, to the HPverb mapping mechanism (FIG. 24).

[0224]FIG. 30 is a flowchart outlining the Post Send and Post ReceiveVerbs mechanism in accordance with the present invention. Step 1 3000:If the verb is a Post Send verb, the HP: validates the input modifiersassociated with the Post Send.

[0225] Step 1.1 3004: If all the modifiers are valid, the HP continuesto step 1.3.

[0226] Step 1.2 3008: If any of the modifiers is invalid, the HP passesback an error immediate verb result, with the specific error, to the HPverb mapping mechanism (FIG. 24).

[0227] Step 1.3 3012: If it is a Send, Raw Send, TOE Send, or Send withSE verb, the HP: creates, for each GL MR, a Redirected_DMA_TCE_entry inthe host Task Control Element (TCE) table (for a description of the TCEmechanism described see co-pending and commonly assigned U.S. patentapplication Ser. No. 10/132,461 (Attorney Docket No. AUS920020065US1))with the access rights set to read; builds the Post Send verb using theinput modifiers passed by the HS; passes the Post Send verb to theIPSOE; and passes back a good immediate verb result to the HP verbmapping mechanism (FIG. 24) (3014).

[0228] Step 1.4 3016: If it is any Send with Register MR verb, the HP:creates, for each Gather List (GL) MR, a Redirected_DMA_TCE_entry in thehost TCE table with the access rights set to the MR access rights;builds the Post Send verb using the input modifiers passed by the HS;passes the Post Send verb to the IPSOE; and passes back a good immediateverb result to the HP verb mapping mechanism (FIG. 24) (3018).

[0229] Step 1.5 3020: If it is a Bind Memory Window verb, the HP:creates, for each GL MR, a Redirected_DMA_TCE_entry in the host TCEtable with the access rights set to the MW access rights; builds thePost Send verb using the input modifiers passed by the HS; passes thePost Send verb to the IPSOE; and passes back a good immediate verbresult to the HP verb mapping mechanism (FIG. 24) (3022).

[0230] Step 1.6 3024: If it is any type of RDMA Write verb, the HP:creates, for each GL MR, a Redirected_DMA_TCE_entry in the host TCEtable with the access rights set to read; builds the Post Send verbusing the input modifiers passed by the HS; passes the Post Send verb tothe IPSOE; and passes back a good immediate verb result to the HP verbmapping mechanism (FIG. 24) (3026).

[0231] Step 1.7 3028: If it is any type of RDMA Read verb, the HP:builds the Post Send verb using the input modifiers passed by the HS;passes the Post Send verb to the IPSOE; and passes back a good immediateverb result to the HP verb mapping mechanism (FIG. 24).

[0232] Step 2 3032: If the verb is a Post Receive (includes RNIC QP, RawQP, and TOE QP Receives) verb, the HP: validates the input modifiersassociated with the Post Receive verb.

[0233] Step 2.1 3036: If all the modifiers are valid, the HP continuesto step 2.3.

[0234] Step 2.2 3040: If any of the modifiers is invalid, the HP passesback an error immediate verb result, with the specific error, to the HPverb mapping mechanism (FIG. 24).

[0235] Step 2.3 3044: The HP: creates, for each Scatter List (SL) MR, aRedirected_DMA_TCE_entry in the host TCE table with the access rightsset to write; builds the Post Receive verb using the input modifierspassed by the HS; passes the Post Receive verb to the IPSOE; and passesback a good immediate verb result to the HP verb mapping mechanism (FIG.24).

[0236]FIG. 31 is a flowchart outlining the Poll CQ, Set CQ Handler, SetAsync Event Handler Verbs mechanism in accordance with the presentinvention. Step 1 3100: If the verb is a Poll CQ verb, the HP: validatesthe input modifiers associated with the Poll CQ (3102).

[0237] Step 1.1 3104: If all the modifiers are valid, the HP continuesto step 1.3.

[0238] Step 1.2 3108: If any of the modifiers is invalid, the HP passesback an error immediate verb result, with the specific error, to the HPverb mapping mechanism (FIG. 24).

[0239] Step 1.3 3112: If there are no Work Completions available on theCQ (3110), the HP passes back the no IPSOE WC to the HP verb mappingmechanism (FIG. 24) (3112).

[0240] Step 1.4 3116: If the Work Completion is associated with a Send,Raw Send, TOE Send, Send with SE verb, any Send with Register MR, orRDMA Write (3114), the HP: destroys, for each GL MR, theRedirected_DMA_TCE_entry in the host TCE table that was created duringthe Post Send verb invocation; and passes back the IPSOE WC to the HPverb mapping mechanism (FIG. 24) (3116).

[0241] Step 1.5 3120: If the Work Completion is associated with a BindMemory Window or RDMA Read (3118), the HP passes back the IPSOE WC tothe HP verb mapping mechanism (FIG. 24) (3120).

[0242] Step 1.6 3124: If the Work Completion is associated with aReceive (includes RNIC QP, Raw QP, and TOE QP Receives), the HP:destroys, for each Scatter List (SL) MR, the Redirected_DMA_TCE_entry inthe host TCE table that was created during the Post Receive verbinvocation; and passes back the IPSOE WC to the HP verb mappingmechanism (FIG. 24).

[0243] Step 2 3128: If it is a Set CQ Handler verb, the HP: validatesthe input modifiers associated with the Post Receive verb.

[0244] Step 2.1 3132: If all the modifiers are valid, the HP continuesto step 2.3.

[0245] Step 2.2 3136: If any of the modifiers is invalid, the HP passesback an error immediate verb result, with the specific error, to the HPverb mapping mechanism (FIG. 24).

[0246] Step 2.3 3138: If a CQ Handler was already created for the HS,the HP: passes back an error verb result to the HP verb mappingmechanism (FIG. 24) (3136).

[0247] Step 2.4 3144: If a CQ Handler was not already created for theHS, the HP: creates a CQ Handler for the HS and passes back a good verbresult to the HP verb mapping mechanism (FIG. 24).

[0248] Step 3 3148: If it is a Set Async Handler verb, the HP: validatesthe input modifiers associated with the Post Receive verb.

[0249] Step 3.1 3152: If all the modifiers are valid, the HP continuesto step 2.3.

[0250] Step 3.2 3156: If any of the modifiers is invalid, the HP passesback an error immediate verb result, with the specific error, to the HPverb mapping mechanism (FIG. 24). If a Async Event Handler was alreadycreated for the HS, the HP: passes back an error verb result to the HPverb mapping mechanism (FIG. 24).

[0251] Step 3.4 3164: If an Async Event Handler was not already createdfor the HS, the HP: creates a CQ Handler for the HS and passes back agood verb result to the HP verb mapping mechanism (FIG. 24).

[0252]FIG. 32 is a flowchart outlining the IPSOE Incoming Ethernet FrameProcessing mechanism in accordance with the present invention. Step 13200: The IPSOE validates that the IP Alias Table contains an entry withan IP address that matches the incoming Ethernet Frame's Destination IPAddress. This step can be combined with step 3.

[0253] Step 2 3204: The network and transport headers of the incomingEthernet Frame are checked to determine the type of QP used in theIPSOE.

[0254] Step 3 3208: If the incoming Ethernet Frame does not containTCP/IP headers or contains a TCP/IP header that has a quintuple(Transport Type, Source IP Address, Destination IP Address, Source PortNumber, and Destination Port Number) which is associated with a Raw orTOE QP, the IPSOE determines if the QP's Receive Work Queue (RWQ) has aWork Queue Element (WQE) that can be used to receive the incoming frame(3210).

[0255] Step 2.1 3212: If the RWQ does not have a WQE available, theIPSOE keeps the frame in the resegmentation buffer or drops it and exitsthis flowchart.

[0256] Step 2.2 3216: If the RWQ has a WQE available, the IPSOE placesthe frame in the WQE and exits this flowchart.

[0257] Step 3 3220: If the incoming Ethernet Frame contains a TCP/IPquintuple (Transport Type, Source IP Address, Destination IP Address,Source Port Number, and Destination Port Number) that is associated witha RNIC QP, the IPSOE determines the type of RNIC Message.

[0258] Step 3.1 3224: If the incoming message is any type of Send, theIPSOE determines if the QP's Receive Work Queue (RWQ) has a Work QueueElement (WQE) that can be used to receive the incoming frame (3226).

[0259] Step 3.1.1 3228: If the RWQ does not have a WQE available, theIPSOE keeps the frame in the resegmentation buffer or drops it.

[0260] Step 3.1.2 3232: If the RWQ has a WQE available, the IPSOE placesthe frame in the WQE.

[0261] Step 3.1.4 3240: If incoming Message is a Send with Invalidate orSend with Invalidate and SE, and the Memory TPT referenced by the STagis associated with the QP that is associated QP that is associated withthe incoming TCP Segment, then the IPSOE invalidates the STag associatedwith the Message.

[0262] Step 3.1.4 3252: If the incoming message is a Send with SE or aSend with Invalidate and SE, generates a CQ Event, and exits thisflowchart.

[0263] Step 3.2.1 3260: If the incoming message is any type of RDMA, theIPSOE performs the following: Uses the index portion of the STag toreference the Memory TPT entry associated with the incoming RDMA Writeand performs the traditional access controls (e.g. validates that theProtection Domain stored in the Memory TPT entry matches the ProtectionDomain Stored in the QP Context). If this access is not valid, the IPSOEperforms the necessary error processing and exits this flowchart.

[0264] Step 3.2.2 3264: The IPSOE determines if the Server Domain (SD)stored in the Memory TPT entry matches the SD stored in the QP Context.If it matches, the IPSOE performs the RDMA operation and exits thisflowchart. If it does not match, the IPSOE performs the necessary errorprocessing and exits this flowchart.

[0265] The description of the present invention has been presented forpurposes of illustration and description, and is not intended to beexhaustive or limited to the invention in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art. The embodiment was chosen and described in order to bestexplain the principles of the invention, the practical application, andto enable others of ordinary skill in the art to understand theinvention for various embodiments with various modifications as aresuited to the particular use contemplated.

What is claimed is:
 1. A method, in a data processing system, for managing a work queue, comprising: receiving a work request; creating a receive work queue entry in a receive work queue in response to receipt of the work request; sending a notification to an Internet Protocol Suite Offload Engine (IPSOE) notifying the IPSOE of the creation of the receive work queue entry; and processing a completion queue entry in a completion queue in response to receiving a notification that the completion queue entry has been created by the IPSOE in response to processing of the receive work queue entry.
 2. The method of claim 1, wherein the completion queue entry includes a work request ID that associates the receive work queue entry to the completion queue entry.
 3. The method of claim 1, further comprising: monitoring a number of entries available in the receive work queue so that valid receive work queue entries are not overwritten.
 4. The method of claim 1, wherein the receive work queue entry is created by an upper layer protocol library in response to a receiving the work request.
 5. A method of sharing an Internet Protocol Suite Offload Engine (IPSOE) between virtual servers of a data processing system, comprising: assigning a first address to a port of the IPSOE; and assigning a second address to the port of the IPSOE, wherein the first address is assigned to a first virtual server, and wherein the second address is assigned to a second virtual server different from the first virtual server.
 6. The method of claim 5, further comprising storing the first and second addresses for the port in a table.
 7. The method of claim 5, further comprising: receiving an incoming frame directed to the port, the incoming frame including a address; performing a lookup in a address table associated with the IPSOE based on the address of the incoming frame; and validating that the address of the incoming frame is one of the first and second addresses associated with the port.
 8. The method of claim 6, wherein each entry in the table is assigned to a specific virtual server.
 9. The method of claim 6, further comprising: assigning an entry in the table for the port to a Send and Receive Queue Pair that access the port.
 10. The method of claim 5, wherein the first and second addresses are Media Access Control (MAC) addresses.
 11. The method of claim 5, wherein the first and second addresses are Internet Protocol (IP) aliases.
 12. The method of claim 5, wherein the first virtual server is associated with a hosted server running an operating system instance on the data processing system, and wherein the second virtual server is associated with a hosting partition of the data processing system.
 13. The method of claim 12, further comprising: generating a software queue pair between the hosted server and the hosting partition.
 14. The method of claim 13, wherein the hosted server places operations on a send queue of the queue pair, and wherein the hosting partition returns results of the operations to the hosted server through a receive queue of the queue pair.
 15. The method of claim 13, further comprising: generating a completion queue for the queue pair; and generating a memory translation and protection table.
 16. The method of claim 15, further comprising: generating a server domain for associating at least one of the queue pair, completion queue and memory translation and protection table with the hosted server.
 17. The method of claim 5, further comprising: generating a first server domain for the first virtual server; and generating a second server domain for the second virtual server, wherein the first server domain associates the first virtual server with a first set of IPSOE resources, and wherein the second server domain associates the second virtual server with a second set of IPSOE resources.
 18. The method of claim 17, further comprising storing the first server domain in a queue pair context, a completion queue context and a Memory Translation and Protection Table entry associated with the first virtual server.
 19. The method of claim 17, further comprising: validating, for an incoming RDMA, that a Memory Translation and Protection Table entry associated with the incoming RDMA contains a same server domain as a queue pair context associated with the incoming RDMA.
 20. The method of claim 17, further comprising: validating, for a receive operation of a receive queue, that a Memory Translation and Protection Table entry associated with the receive operation contains a same server domain value as a queue pair context associated with the receive queue.
 21. The method of claim 17, further comprising: validating, for any of a Send, RDMA Write, or Bind operation of a send queue, that a Memory Translation and Protection Table entry associated with the operation contains a same server domain value as a queue pair context associated with the send queue.
 22. The method of claim 17, further comprising: validating, for completed work queue entries, that a completion queue context and queue pair that completed the work queue entry contain a same server domain value.
 23. The method of claim 5, further comprising: providing a IPSOE Resource Management Table (RMT) for assigning IPSOE resources to the first and second virtual servers.
 24. The method of claim 23, further comprising: storing in the RMT, for each physical IPSOE resource, a maximum value of the physical IPSOE resource.
 25. The method of claim 23, further comprising: storing in the RMT, for each variable physical IPSOE resource of the physical IPSOE resources, a total value of the variable physical IPSOE resource that has already been allocated.
 26. The method of claim 23, further comprising: storing, for each of the first and second virtual servers, a maximum amount of variable IPSOE resources, an amount of a fixed IPSOE resource that are assigned to the virtual server, and a total number of variable IPSOE resources that are currently in use by the virtual server.
 27. The method of claim 26, further comprising: providing, for each of the first and second virtual server, a mechanism for preventing the total number of variable IPSOE resources that are currently in use by the virtual server from exceeding the maximum variable IPSOE resources that are assigned to the virtual server.
 28. The method of claim 1, further comprising: providing a Selective Acknowledgment (SACK) table for storing information regarding incoming data packets destined for a receive queue of the IPSOE and for generating TCP/IP SACKs.
 29. A computer program product in a computer readable medium for sharing an Internet Protocol Suite Offload Engine (IPSOE) between virtual servers of a data processing system, comprising: first instructions for assigning a first address to a port of the IPSOE; and second instructions for assigning a second address to the port of the IPSOE, wherein the first address is assigned to a first virtual server, and wherein the second address is assigned to a second virtual server different from the first virtual server.
 30. An apparatus for sharing an Internet Protocol Suite Offload Engine (IPSOE) between virtual servers of a data processing system, comprising: means for assigning a first address to a port of the IPSOE; and means for assigning a second address to the port of the IPSOE, wherein the first address is assigned to a first virtual server, and wherein the second address is assigned to a second virtual server different from the first virtual server. 